Organism sample observation device

ABSTRACT

A biological specimen observation apparatus whereby observation of a biological specimen can be performed accurately. In macro observation, a biological change region is extracted from a macro image, a micro observation point corresponding to an extracted biological change region is registered, and an object for tracking is identified. In micro observation, it is judged from the micro image whether or not biological change has continued in the biological change region at the micro observation point, and the registered micro observation point is updated on the basis of this judgment result. It is possible to carry out both macro observation for detecting a biological change region and micro observation for observing the progress of growth of a partial minute region where biological change has been exhibited, and to carry out accurate observation of a biological specimen.

TECHNICAL FIELD

The present invention relates to a biological specimen observationapparatus.

BACKGROUND ART

Conventionally, a biological specimen observation apparatus whichcarries out observation of the progress of growth of a cultivatedbiological specimen is known.

In an apparatus of this kind, a method using time lapse observation isgenerally employed as a method for acquiring observation images of abiological specimen, but the following techniques have been proposed inthe prior art, for example.

Patent Document 1 describes technology which enables the start ofoperation of an automatic time lapse observation when it has beendetected that there has been a change in a biological specimen which isan observation object, after which, if it is detected that change in thebiological specimen has stopped, the operation of automatic time lapseobservation is halted.

-   Patent Document 1: Japanese Patent Application Publication No.    2004-309719

However, in the case of a technique which is disclosed in PatentDocument 1, an image for detecting change in a biological specimen andan image obtained by time lapse observation started after detection areset to the same magnification rate at all times, and therefore it is notpossible to observe a region where the biological specimen has changedat a high magnification rate, and hence there is a problem in that thebiological specimen cannot be observed accurately.

SUMMARY OF THE INVENTION

The present invention was devised in view of these circumstances, andmakes it possible to carry out accurate observation of a biologicalspecimen so as to obtain an observation image at a suitablemagnification rate in accordance with the detection circumstances of thebiological specimen.

A first biological specimen observation apparatus according to thepresent invention is a biological specimen observation apparatus whichobserves temporal change in a biological specimen, having: macro imageacquisition means for acquiring a macro image by capturing an image of amacro region of a broad range of the biological specimen, while timelapse observation is performed; biological change region extractingmeans for extracting a biological change region, which is a region ofchange in the biological specimen, from the acquired macro image; microobservation point setting means for registering a micro observationpoint corresponding to the extracted biological change region; microimage acquisition means for capturing an image of a micro region ofchange in the biological specimen identified at the micro observationpoint, while time lapse observation is performed; and judgment means forjudging whether or not biological change has continued in the biologicalchange region at the micro observation point, from the acquired microimage; wherein the micro observation point setting means updates theregistered micro observation point on the basis of a judgment result bythe judgment means.

A second biological specimen observation apparatus according to thepresent invention is a biological specimen observation apparatus whichobserves temporal change in a biological specimen, having: macro imageacquisition means for acquiring a macro image by capturing an image of amacro region of the biological specimen, while time lapse observation isexecuted; micro observation point setting means for identifying a microobservation point where micro observation is to be performed in themacro region of the biological specimen, on the basis of the macroimages of a time series obtained by the macro image acquisition means;and micro image acquisition means for performing time lapse observationand acquiring micro images of the biological specimen at the microobservation point identified by the micro observation point settingmeans.

A third biological specimen observation apparatus according to thepresent invention is a biological specimen observation apparatus whichobserves temporal change in a biological specimen, having: macro imageacquisition means for acquiring a macro image by capturing an image of amacro region of a broad range of the biological specimen, while timelapse observation is performed; and micro image acquisition means foracquiring a micro image by capturing an image of a micro region in themacro region of the biological specimen, while time lapse observation isperformed; wherein temporal change in the biological specimen isanalyzed from the acquired macro image and micro image by the macroimage acquisition means and the micro image acquisition means.

According to the present invention, it is possible to carry outobservation of a biological specimen accurately.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a front diagram showing the general composition of abiological specimen observation apparatus to which the present inventionis applied;

FIG. 2 is a diagram showing a detailed example of the composition of anobservation unit;

FIG. 3 is a flowchart which describes an observation schedule settingprocess;

FIG. 4 is a diagram which describes a schedule setting process in thecase where the priority mode is set to micro observation;

FIG. 5 is a diagram which describes a schedule setting process in thecase where the priority mode is set to macro observation;

FIG. 6 is a flowchart describing a time lapse observation process;

FIG. 7 is a flowchart describing a process for extracting a biologicalchange region from a tiling image;

FIG. 8 is a diagram describing tiled image generating means;

FIG. 9 is a flowchart describing an observation process of a peripheralarea of a micro observation point in micro observation; and

FIG. 10 is a diagram showing an example of a peripheral image obtainedin the peripheral area of a micro observation point.

MODE OF IMPLEMENTING THE INVENTION

Below, an embodiment of the present invention is described withreference to the drawings.

FIG. 1 is a front view diagram showing the general composition of abiological specimen observation apparatus to which the present inventionis applied.

In FIG. 1, the solid lines indicate the structure of parts which arevisible in external view, and the broken lines indicate the structure ofinternal parts which are not visible in external view.

As shown in FIG. 1, the biological specimen observation apparatus 1includes a first frame 11 in which cultivation of a biological specimenis carried out, and a second frame 12 which constitutes a controlapparatus. The first frame 11 is used in a state of being mounted on topof the second frame 12.

A thermostatic chamber 21 which is covered with a heat insulating memberis formed inside the first frame 11. This thermostatic chamber 21 isconnected to the exterior by means of a front surface opening 23 whichis formed on the front surface of the first case 11 (front surface door22), and an inward and outward conveyance port 24 formed in theleft-hand side face as viewed from the front surface of the first case11.

The thermostatic chamber 21 is provided with, for example, a temperaturecontrol mechanism consisting of a temperature adjustment apparatus, orthe like, employing a Peltier element, a humidity control mechanismconsisting of a spraying apparatus, or the like, which sprays a mist, agas control mechanism consisting of a gas introduction unit, or thelike, which is connected to an external carbon dioxide gas cylinder, anenvironment sensor which determines the cultivation environment of thebiological specimen in the internal space, and the like (all omittedfrom the drawings). By this means, the interior of the thermostaticchamber 21 is sealed in order to maintain the cultivation environment ofthe biological specimen during cultivation of the biological specimen,and is kept at a uniform temperature by circulating the air, forexample, thereby maintaining the interior of the thermostatic chamber 21at a temperature of 37° C., a humidity of 90%, and a carbon dioxideconcentration of 5%, and so on.

Moreover, a stocker 25, a vessel conveyance mechanism 26 and anobservation unit 27 are accommodated inside the thermostatic chamber 21of the first frame 11.

The stocker 25 is divided in the up/down direction by a plurality ofshelves, whereby culture vessels 15 (FIG. 2) can be accommodatedhorizontally therein.

The vessel conveyance mechanism 26 is provided with a conveyance armsection which supports a holder, and various mechanisms (notillustrated) for conveying culture vessels 15. The vessel conveyancemechanism 26 is able to move a holder supported on the conveyance armsection, in the vertical direction (Z direction) or the horizontaldirections (X and Y directions), or to rotate the holder through 180degrees about the Z axis.

Next, the observation unit 27 is described with reference to FIG. 2.

A front surface diagram of the observation unit 27 is shown on theright-hand side of FIG. 2, and a side face diagram of the observationunit 27 is shown on the left-hand side of FIG. 2.

As shown in FIG. 2, the observation unit 27 is constituted by atransmission light illumination unit 41, an epifluorescent illuminationunit 42, a specimen platform 43 and an observation unit 44.

The transmission light illumination unit 41 is formed in an arm shape soas to extend upwards from the side portion of the specimen platform 43and then extend over the culture vessel 15 which is loaded on thespecimen platform 43. A transmission light LED (Light Emitting Diode) 51and a transmission light optical system 52 are accommodated inside thetransmission light illumination unit 41. The transmission light LED 51emits light of a prescribed wavelength region. The light from thetransmission light LED 51 is irradiated from the upper side onto aculture vessel 15 which is loaded on the specimen platform 43, via thetransmission light optical system 52.

The epifluorescent illumination unit 42 is constituted by fluorescenceLEDs 53 a to 53 c, and a fluorescence optical system 54. Thefluorescence LEDs 53 a to 53 c (simply called “fluorescence LEDs 53”below) respectively emit light of different wavelengths, in other words,light of a prescribed wavelength (excitation light) corresponding tofluorescent material in the biological specimen in the culture vessel 15is emitted. The excitation light from the transmission light LED 53 isirradiated from the lower side onto a culture vessel 15 which is loadedon the specimen platform 43, via a fluorescent light optical system 54and an object lens 55.

The specimen platform 43 is made of a light transmitting material andlight from the transmission light illumination unit 41 and light fromthe fluorescent material included in the biological specimen which isexcited by the light from the epifluorescent illumination unit 42 inintroduced into the observation unit 44. Furthermore, an object lens 55which condenses light introduced to the observation unit 44, and a stage56 which moves a culture vessel 15 in the vertical direction or thehorizontal direction in order to capture an image of a prescribedlocation of the biological specimen, and the like, are also provided onthe specimen platform 43. The object lens 55 is constituted by aplurality of object lenses having different magnification rates (forexample, 2×, 4×, 10×, 20×, 40×, . . . , etc.), and hence the observationmagnification rate can be switched appropriately in accordance with theobservation circumstances.

The observation unit 44 is constituted by an imaging unit 57 and animage processing unit 58. The imaging unit 57 has an imaging element,such as a CCD (Charge Coupled Device), or the like. On an image surfaceof the CCD, an image produced by the light from the transmission lightirradiation unit 41 and an image produced by light from the fluorescentmaterial contained in the biological specimen which is excited by lightfrom the epifluorescent illumination unit 42 are formed by an imageforming optical system.

The image processing unit 58 obtains image data represented by a digitalsignal, by applying analog signal processing for amplifying the analogimage signal from the imaging unit 57, for example, and then convertingthe signal from analog to digital (A/D).

Returning to FIG. 1, besides a portion of the observation unit 27described above, a control unit 31 is also accommodated inside thesecond frame 12 on which the first frame 11 is mounted.

The control unit 31 controls the operation of the respective units ofthe biological specimen observation apparatus 1. More specifically, thecontrol unit 31 executes adjustment of the environmental conditionsinside the thermostatic chamber 21, inward and outward conveyance of theculture vessel 15 into and out from the thermostatic chamber 21,observation of the biological specimen inside the culture vessel 15,conveyance of a culture vessel 15 inside the thermostatic chamber 21,and the like, in accordance with an observation schedule or directinstructions based on operations performed by a user.

The observation schedule can be set, for example, by means of a setupscreen which is displayed on a display panel 32 which displays variousinformation processed by the control unit 31, and is input by inputmeans, such as a keyboard, or the like, which is connected to thecontrol unit 31. By this means, the observation position, theobservation magnification rate, the imaging schedule, and the like, areset up for each sample.

Furthermore, an observation information memory unit 33 is providedinside the control unit 31, and this observation information memory unit33 stores and accumulates image data corresponding to observation imagesobtained by time lapse observation which are supplied from the imageprocessing unit 58. The image data stored in the observation informationmemory unit 33 is recorded in association with identificationinformation for the culture vessel 15 and index information indicatingthe date and time of image capture. Furthermore, information of varioustypes relating to the observation of the biological specimen, and ahistory of change in the environmental conditions (temperature,humidity, carbon dioxide concentration, etc.) inside the thermostaticchamber 21, can also be recorded in the observation information memoryunit 33.

The image analysis unit 34 carries out image analysis by performingprescribed image analysis processing on the image data accumulated inthe observation information memory unit 33. The image analysis unit 34supplies the image analysis results to the observation control unit 35.

The observation control unit 35 controls the operation of theobservation unit 27 in accordance with the observation schedule ordirect instructions based on operations performed by a user.Furthermore, the observation control unit 35 controls the operation ofthe observation unit 27 which carries out observation of a biologicalspecimen in a culture vessel 15, on the basis of the image analysisresults from the image analysis unit 34.

The control unit 31 comprises communications means (not illustrated)which are compliant with a prescribed wireless or wired communicationsstandard, and is able to send and receive data to and from externaldevices, such as a personal computer, via a network. By this means, itis also possible to observe a biological specimen, change the imagingconditions setup, and check the environment in the thermostatic chamber21 and the observation images, from a personal computer which ispositioned in a remote location, using a network.

Furthermore, in the biological specimen observation apparatus 1,observation of two patterns, namely, macro observation and microobservation, are carried out as observation by time lapse.

Here, macro observation means observation using an object lens having alow magnification rate (for example, 2×, 4×) which can obtain anobservation image corresponding to a broad region on the biologicalspecimen, as the object lens 55. Below, an observation image obtained bymacro observation is called a macro image. Furthermore, microobservation is observation using an object lens having a highermagnification rate than macro observation (for example, 4×, 10×, 20×,40×). Below, an observation image obtained by micro observation iscalled a micro image.

In other words, in the observation unit 27, when observation of a broadrange of the biological specimen is to be carried out, then an objectlens of low magnification rate, from the plurality of object lenses 55,is arranged in the optical path of the light of the biological specimenin the culture vessel 15, and macro observation is carried out. On theother hand, when observation of a particular range of the biologicalspecimen is to be carried out, then an object lens of high magnificationrate, from the plurality of object lenses 55, is arranged in the opticalpath of the light of the biological specimen in the culture vessel 15,and micro observation is carried out. By carrying out macro observationand micro observation, macro images and micro images having differentobservation magnification rates are accumulated as observation images inthe observation information memory unit 33.

Next, the operation of the biological specimen observation apparatus 1which carries out macro observation and micro observation will bedescribed.

As described above, in the biological specimen observation apparatus 1,observation of temporal change in a biological specimen which is beingcultivated is carried out in accordance with an observation schedule,and this observation schedule is set by an operation performed by theuser. Therefore, firstly, a process for setting up an observationschedule for macro observation and micro observation will be describedwith reference to the flowchart in FIG. 3.

This setup process is executed, for example, by the control unit 31, inaccordance with an input operation by the user made to a setup screenfor setting up observation schedules of various types which is displayedon the display panel 32.

In step S11, when information relating to the observation magnificationrate for macro observation and settings for fluorescent observation hasbeen input, the macro observation condition settings are implemented onthe basis of these parameters. The observation magnification rate isset, for example, to two times or four times, or the like. Furthermore,in the present embodiment, a case where fluorescent observation iscarried out is described, but when fluorescent observation is notcarried out, there is not need to set information relating to thesettings for fluorescent observation, and only the observationmagnification rate is set.

By means of these parameters, the time required for one macroobservation (called the macro observation prescribed time T) isdetermined.

In step S12, when information relating to the observation-magnificationrate for micro observation and settings relating to fluorescentobservation, the number of frames, and the maximum number of observationpoints, has been input, the micro observation condition settings areimplemented on the basis of these parameters. The observationmagnification rate is set, for example, to 10 times or 20 times, or thelike. In other words, as described above, the observation magnificationrate in micro observation is set so as to be a higher magnification ratethan the observation magnification rate for macro observation.

Furthermore, the frame number set as the micro observation condition isthe number of micro images acquired within a prescribed time period. Ifthe number of frames is set to a large number, then the data volume ofthe acquired micro images becomes larger, but since the number of microimages acquired per unit time increases, then it is possible to carryout more detailed micro observation. Furthermore, the maximum number ofobservation points is the maximum number of positions which can be setfor observation as a region where there is change in the biologicalspecimen, in a micro image (hereinafter, called micro observationpoints). By setting this maximum number to a large number, it ispossible to observe change in the biological specimen at a greaternumber of points. Furthermore, although the micro observation prescribedtime t, which is described below, cannot be determined until the numberof observation points is designated, the number of observation points isdetermined on the basis of biological change regions which are extractedfrom a macro image, and therefore cannot be determined at this stage.Therefore, the micro observation prescribed time t is determined byusing this maximum number of observation points. In this way, these twoparameters are parameters which are specific to micro observation andtherefore do not have to be set as macro observation conditions.

By means of these parameters, the time required for one microobservation operation (called the micro observation prescribed time t)is determined.

In step S13, when information is input indicating which of macroobservation and micro observation is prioritized, then priority modeselection is implemented. The subsequent processing is different,depending on which type of observation is prioritized and correspondingdetails of this are described below.

In step S14, when information has been input in relation to the intervalbetween the nth macro observation and the n+1th macro observation(hereinafter, interval period ΔT) and the number of macro images (numberof observations) which can be obtained during the observation period ofmacro observation, then an observation schedule condition setup formacro observation is implemented on the basis of these parameters. Thevalue of ΔT is set so as to be a longer interval than the macroobservation prescribed time T. In other words, ΔT is set so as tosatisfy the relationship in Formula (1) below.

ΔT>T  (1)

In step S15, when information is input in relation to the intervalbetween the mth micro observation and the m+1th micro observation(called the interval period Δt), then setup of the observation scheduleconditions for micro observation is implemented on the basis of thisparameter. Similarly to Formula (1) relating to ΔT described above, Δtis set so as to be a longer interval than the micro observationprescribed time t, and is set so as to satisfy the relationship inFormula (2) below.

Δt>t  (2)

In step S16, satisfying the relationship ΔT>t+T means a condition forperforming micro observation after macro observation, and therefore itis judged whether or not this condition is satisfied. Consequently, instep S16, if it is judged that the relationship ΔT>t+T is not satisfied,then the procedure returns to step S14, and the values of ΔT, T and twhich were described above are reset. On the other hand, if it is judgedthat the relationship ΔT>t+T is satisfied, then the processing advancesto step S17.

At step S17, it is judged whether or not the priority mode selected inthe processing step S13 is micro observation mode. If it is judged instep S17 that macro observation is prioritized, then it is judged thatthe condition for carrying out micro observation after macro observationis satisfied, (“Yes” at step S16), and the processing then advances tothe setup process in step S20. On the other hand, if it is judged thatthe micro observation is prioritized, then the processing advances tostep S18.

In step S18, satisfying the relationship Δt>t+T means a condition forperforming macro observation after micro observation, and therefore itis judged whether or not this condition is satisfied. Consequently, ifit is judged at step S18 that the relationship Δt>t+T is satisfied, thenit is judged that micro observation is prioritized (“Yes” at step S17),and the processing advances to the setup process in step S20. On theother hand, if it is judged that the relationship Δt>t+T is notsatisfied, then the processing advances to step S19.

At step S19, it is judged whether or not to change the priority mode inaccordance with input operations by the user, and at step S19, if it isjudged that the priority mode is changed, then the processing returns tostep S13, and a reselection process for the priority mode is carriedout. On the other hand, if it is judged at step S19 that the prioritymode is not to be changed, then the processing returns to step S15, theobservation schedule condition setup for micro observation is carriedout again, and the values of T, Δt and t described above are reset.

Thereafter, if the judgment in step S17 is “No” or the judgment in stepS18 is “Yes”, then the processing advances to step S20.

In step S20, when information relating to the start timing of time lapseobservation is input, the observation start time is set and theobservation schedule is established. Even though an observation schedulehas been established, the processing which is executed differs between acase where micro observation has been selected and a case where macroobservation has been selected, and therefore the processing which isexecuted in each type of observation depending on the priority mode willbe described individually here while referring to FIG. 4 and FIG. 5.

Firstly, processing in a case where micro observation is selected as thepriority mode is described with reference to FIG. 4. In FIG. 4 and FIG.5, the upper part of the diagram shows an observation schedule of macroobservation which is registered in the setup process in step S20, andmacro observation requiring observation times of T₁, T₂, T₃, . . . ,T_(n) is carried out at intervals of ΔT. Furthermore, the lower part ofeach diagram similarly shows an observation schedule for microobservation which is registered in the setup processing in step S20, andmicro observation requiring observation times of t₁, t₂, t₃, . . . ,t_(m) is carried out at intervals of Δt.

In the observation schedule in FIG. 4A, micro observation isprioritized, and therefore the relationship Δt>t+T is satisfied, butthere is overlap between the macro observation time T₂ in the upper partand the micro observation time t₃ in the lower part and therefore it isnot possible to carry out both types of observation in this state. Inthis case, the macro observation time T₂ in the upper part is staggeredto a macro observation time T₂′ as shown in the observation schedule inFIG. 4B, so as to delay the start time of macro observation. By thismeans, micro observation at micro observation time t₃ is carried outpreferentially, and when this micro observation has ended, macroobservation is carried out immediately at macro observation time T₂′.

In other words, when micro observation is prioritized, Δt is uniform,and therefore ΔT is changed to alter the observation schedule in such amanner that macro observation is carried out after the end of microobservation, as shown by FIG. 4B.

Next, processing in a case where macro observation is selected as thepriority mode is described with reference to FIG. 5.

In the observation schedule in FIG. 5A, macro observation isprioritized, and therefore the relationship ΔT>t+T is satisfied, butsimilarly to FIG. 4A, there is an overlap between the macro observationtime T₂ in the upper part and the micro observation time t₃ in the lowerpart. In this case, as indicated by the dotted line in FIG. 5B, themicro observation time t₃ in the lower part which overlaps with themacro observation time T₂ in the upper part is erased, and macroobservation at the macro observation time T₂ is carried outpreferentially, without carrying out micro observation at microobservation time t₃. Furthermore, rather than omitting the microobservation which has an overlapping observation time, by staggering themicro observation time t₃ of the lower part to micro observation timet₃′ so as to delay the start time of micro observation, it is possibleto carry out micro observation at the micro observation time t₃′immediately after the macro observation at macro observation time T₂ hasended.

In other words, when macro observation is prioritized, ΔT is uniform,and therefore Δt is changed to alter the observation schedule in such amanner that micro observation is carried out after the end of macroobservation, as shown by FIG. 5B.

As described above, a setup process for an observation schedule based onrespective priority modes is carried out in step S20 and processing thenterminates.

There follows a description of processing for macro observation andmicro observation based on time lapse observation which is carried outin accordance with the observation schedule described above, withreference to the flowchart in FIG. 6.

At step S31, the observation unit 27 carries out macro observation ofthe biological specimen in accordance with control by the observationcontrol unit 35 based on the observation schedule. The macro imagesobtained by macro observation are accumulated in the observationinformation memory unit 33.

At step S32, the image analysis unit 34 extracts a region of change inthe biological specimen (hereinafter, called a “biological changeregion”) by applying prescribed image analysis processing to the macroimages accumulated in the observation information memory unit 33. Forexample, a macro image which is a fluorescent image is obtained bycarrying out fluorescent observation in the observation unit 27, butthis macro image includes a luminous point originating from thefluorescence color included in the biological specimen, and thisluminous point is extracted by means of a suitable threshold value byimage analysis processing. For example, change in the biologicalspecimen, such as formation of a colony, deformation of cells oroccurrence of fluorescence, or the like, appears as change in theluminous point of the macro images which are obtained over time by timelapse observation (for example, change in the value (RGB value) ofpixels (pixels of interest) at the same position respectively in then−1th macro image (the macro image at macro observation time T_(n-1))and the nth macro image (the macro image at macro observation timeT_(n))), and therefore the region where this change has occurred isextracted as a biological change region and is supplied to theobservation control unit 35.

At step S33, if the biological change region is a newly extractedbiological change region, of the biological change regions extractedfrom the macro images, then the observation control unit 35 newlyregisters a micro observation point which specifies the centralcoordinates of the biological change region (for example, the XYZcoordinates of the stage 56 or the object lens 55), whereas if thebiological change region is an already registered biological changeregion, then the observation control unit 35 sets a micro observationpoint by updating the micro observation point which specifies thatbiological change region. The information (XYZ coordinates) relating tothis micro observation point is stored and set in the observationinformation memory unit 33, for instance.

In step S34, the observation control unit 35 judges whether or not thereis a micro observation point, on the basis of the coordinates of theestablished micro observation point.

In step S34, if a biological change region has not been extracted and itis judged that there does not exist even one micro observation point,then at step S35, the observation control unit 35 judges whether or notthe observation end time has been reached, on the basis of theobservation schedule. If it is judged at step S35 that the observationend time has been reached, then the time lapse observation process isterminated. On the other hand, if it is judged at step S35 that theobservation end time has not been reached, then at step S36, theobservation control unit 35 keeps the observation unit 27 waiting untilthe next macro observation start time. In this case, it is also possibleto receive the input of registration or change of coordinates of themicro observation point in accordance with operations performed by theuser; for example, it is possible to set a desired region which has notbeen extracted in the image analysis processing described above, as amicro observation point.

Thereupon, at step S36, when the macro observation start time isreached, the processing returns to step S31 and the macro observationprocessing in steps S31 to S36 described above is repeated either untilit is judged that there exists a micro observation point or until it isjudged that the observation end time has been reached.

If it is judged at step S34 that there is a micro observation point,then at step S37, the observation control unit 35 refers to theinformation relating to micro observation points stored in theobservation information memory unit 33 and judges whether or not therehas been an increase or decrease in the number of micro observationpoints.

If it is judged in step S37 that there has been an increase or decreasein the number of micro observation points, then at step S38, theobservation control unit 35 carries out updating of the observationschedule for micro observation, and information relating to the microobservation points which have been increased or decreased is reflectedin the observation information memory unit 33. By this means, microobservation is carried out at newly found micro observation points.

On the other hand, if it is judged at step S37 that there has been noincrease or decrease in the micro observation points, then there is noneed to update the observation schedule for micro observation, andtherefore the step S38 is skipped and processing advances to step S39.

At step S39, the observation control unit 35 causes the observation unit27 to wait until the micro observation start time is reached. In thiscase, in the observation unit 27, preparatory operations for microobservation of the micro observation points are carried out, such as anoperation for switching from an object lens of low magnification ratefor macro observation to an object lens of high magnification rate formicro observation, or an operation by the stage 56 for moving theculture vessel 15 in the vertical direction or horizontal direction forthe purposes of time lapse observation of the micro observation points,for instance. Furthermore, in this case, it is possible to receive inputof the registration or change of a micro observation point in accordancewith operations by the user, and to set a new micro observation point.

Thereupon, when the micro observation start time is reached in step S39,then at step S40, the observation unit 27 carries out micro observationof the micro observation point under the control of the observationcontrol unit 35 based on the observation schedule. The micro imagesobtained by micro observation are accumulated in the observationinformation memory unit 33.

These micro images are images obtained by microscopic observation of aportion of a macro image in which the whole of the biological specimencan be observed over a broad range, and correspond to a partial minuteregion of a biological change region which has been extracted by macroobservation. In other words, a micro image enables observation of finerchanges in a partial minute region of the biological specimen, comparedto a macro image, and therefore the biological specimen can be observedin an accurate fashion.

In step S41, the image analysis unit 34 carries out prescribed imageanalysis processing on the micro images accumulated in the observationinformation memory unit 33, and judges whether change in the biologicalspecimen has occurred continuously at the micro observation pointspecified by the processing in step S33, or whether there has been nochange.

In step S42, the observation control unit 35 updates the registeredmicro observation points in accordance with the state of change in thebiological change regions at the micro observation points.

More specifically, if it is judged that there is a biological change,then the coordinates of the micro observation points registered(updated) by the macro observation continue to be updated in microobservation as well. In other words, in micro observation, the detailsof the aspect of the biological specimen after change are tracked bymeans of high-definition micro images which are identified by microobservation points which have been detected by macro observation.

Furthermore, conversely, if it is judged that there is no biologicalchange, then the observation control unit 35 removes the microobservation point where tracking does not need to be continued, and theacquisition of micro images at this micro observation point is halted.For example, if a colony which is registered as a micro observationpoint does not satisfy a prescribed standard value based on prescribedimage analysis processing and has not been judged to be an iPS colony,then this point is removed from the micro observation points. Bysuspending observation at points such as this, it is possible to avoidunnecessary processing in the time lapse observation process.

In step S43, the observation control unit 35 judges whether or not theobservation end time has been reached, on the basis of the observationschedule. If it is judged at step S43 that the observation end time hasbeen reached, then the time lapse observation process is terminated. Onthe other hand, if it is judged at step S43 that the observation endtime has not been reached, then at step S44, the observation controlunit 35 judges whether or not either the next macro observation starttime or the next micro observation start time is near, on the basis ofthe observation schedule.

At step S44, if the next micro observation start time is first, then theprocedure returns to step S39, and the micro observation processing insteps S39 to S44 described above is repeated until either it is judgedthat the next macro observation start time is first or it is judged thatthe observation end time has been reached. Furthermore, in this case, ifa plurality of micro observation points are set and registered, thensimilar micro observation is carried out for the other micro observationpoints as well.

On the other hand, if it is judged at step S44 that the next macroobservation start time is first, then the procedure returns to step S36.If the macro observation start time has been reached, then theprocessing returns to step S31, detection of a biological change regionis carried out over a broad range as macro observation processing, andthe registration and update setting processes for the micro observationpoints described above are carried out again.

As stated previously, it is possible to carry out both macro observationfor detecting a biological change region and micro observation forobserving the progress of growth of a partial minute region wherebiological change has been observed, and therefore it is possible todetect biological change over a broad region, as well as being able toperform detailed observation of a minute region which has changed.

In this way, in the biological specimen observation apparatus 1, it ispossible to carry out time lapse observation based on a priority modewhich is suited to the objectives of the sample observation.

A macro image which is captured by time lapse observation in macroobservation does not have to be captured in one instant, but may be animage produced by pasting together a plurality of captured images usinga so-called tiling technique, (hereinafter, called a “tiled image”).

Next, the process of extracting a biological change region from a tiledimage is described with reference to the flowchart in FIG. 7.

More specifically, steps S71 to S74 in FIG. 7 correspond to macroobservation processing which is executed in steps S31 and S32 in FIG. 6,and the processing from step S74 onwards is similar to the processing instep S33 onwards in FIG. 6.

In step S71, the observation control unit 35 specifies a number ofimages which are to be acquired from a designated observation area, whena region to be observed by macro observation (observation area) has beendesignated by an operation performed by the user.

At step S72, the observation unit 27 carries out macro observation ofthe biological specimen in accordance with control by the observationcontrol unit 35. In this macro observation, if an acquired image isrepresented as captured image I_(i,j) (i: number of scans in thevertical direction; j: number of scans in the horizontal direction),then as shown in FIG. 8, eight captured images I₁₁ to I₁₈ are acquiredby the first scan in the horizontal direction. Thereafter, by performingscanning which includes eight repetitions of a shift operation in thevertical direction, in such a manner that the captured images do notoverlap (alternatively, it also possible to adopt a pasting region forpasting together with other captured images which are adjacent in thelateral or longitudinal directions), captured images I₂₁ to I₂₈,captured images I₃₁ to I₃₈, . . . , and captured images I₈₁ to I₈₈, areacquired sequentially, and 8×8 captured images I_(i,j) in FIG. 8 areobtained corresponding to the designated observation area.

In step S73, the image processing unit 58 generates one tiled image fromthe 8×8 captured images in FIG. 8, for instance, by tiling together thecaptured images obtained by scanning. This tiled image corresponds tothe macro image shown in FIG. 6.

In step S74, similarly to step S32 in FIG. 6, a biological change regionis extracted from the tiling image forming a macro image, by the imageanalysis unit 34, and thereafter processing similar to that in step S33in FIG. 6 is carried out.

In this way, even in cases where a image covering a broad region cannotbe obtained instantaneously, by tiling a plurality of images to form aprescribed size, it is possible to obtain an image for a broad region,which can be used as a macro image.

In micro observation, an image of a biological change region extractedby macro observation is acquired, but there may be cases where one microimage obtained from a registered micro observation point only includes aportion of a biological change region. In micro observation in casessuch as this, it is not possible to obtain an image of the whole of thebiological change region, and hence there is a possibility that thebiological specimen cannot be observed accurately. Therefore, in thepresent embodiment, if the whole of a biological change region is notcontained completely within one micro image, then an image of the wholeregion is obtained by combining together images obtained from aperipheral area of a micro observation point (called peripheral imagesbelow).

FIG. 9 is a flowchart describing an observation process of a peripheralarea of a micro observation point.

More specifically, steps S91 to S95 in FIG. 9 correspond to microobservation processing which is executed in step S40 in FIG. 6, and theprocessing from step S95 onwards is similar to the processing in stepS41 onwards in FIG. 6.

In this example, in order to make the description easy to understand, asshown in FIG. 10, nine images I₁₁ to I₃₃ are required in order to obtainan image of the whole range of the biological change region (the blackregion in FIG. 10), and the micro image obtained by the microobservation described above is described as a micro image I₂₂ in centralpart thereof.

At step S91, the observation unit 27 carries out micro observation of amicro observation point, similarly to step S40 in FIG. 6. In thisexample, a micro image I₂₂ shown in FIG. 10 is obtained by microobservation of the micro observation point.

At step S92, the image analysis unit 34 applies prescribed imageanalysis processing to the acquired micro image, for instance, and fromthis analysis result, it is judged whether or not it has been possibleto acquire an image of the whole of the biological change region. Instep S92, if it is judged that the whole of the biological change regionis contained within the micro image, then it is not necessary to processthe micro image, and therefore the procedure advances to step S41 inFIG. 6. The processing in this case is similar to the processingdescribed in relation to FIG. 6 above.

On the other hand, if it is judged at step S92 that the whole of thebiological change region is not contained within the micro image, thenat step S93, the observation unit 27 carries out micro observation inthe peripheral area of the micro observation point as well. By thismicro observation, a peripheral image including the biological changeregion is obtained from the region of the peripheral area of the microobservation point. For example, as shown in FIG. 10, when a micro imageI₂₂ has been obtained by micro observation of the micro observationpoint, then eight peripheral images I₁₁ to I₁₃, I₂₁, I₂₃, I₃₁ to I₃₃ areobtained by further micro observation of the peripheral area. In otherwords, if the one micro image I₂₂ and the eight peripheral images I₁₁ toI₁₃, I₂₁, I₂₃, I₃₁ to I₃₃, are viewed as a single image, then an imageincluding the whole of the biological change region is obtained.

Thereupon, in step S94, it is judged whether or not to carry out tiling,and if tiling is not to be carried out, then processing advances to stepS41 in FIG. 6, whereas if tiling is to be carried out, the processingadvances to step S95.

At step S95, similarly to the tiling process in step S73 in FIG. 7, theimage processing unit 58 generates one tiled image by tiling the microimage I₂₂ obtained by micro observation of the micro observation point,and the peripheral images I₁₁ to I₁₃, I₂₁, I₂₃, I₃₁ to I₃₃ obtained bymicro observation of the peripheral area of the micro observation point.Since the tiled image is regarded as one micro image, then a micro imageincluding the whole of the biological change region is produced.Thereupon, the processing advances to step S41 in FIG. 6 and biologicalchange judgment processing is carried out.

In this way, if it is not possible for the whole range of a biologicalchange region to be contained in one micro image, it is possible toacquire an image of the whole of the biological change region byacquiring and tiling peripheral images. By this means, it is possible tocheck the whole of the biological specimen which is an object ofobservation, and therefore it is possible to carry out accurateobservation.

As described above, according to the present embodiment, change in thebiological specimen which is the object of tracking by macro observationis extracted over a broad range, whereupon a biological change regionidentified by macro observation can be observed microscopically, andtherefore it is possible to carry out observation accurately.

More specifically, in the present embodiment, it is possible to carryout time lapse observation of a region where an observation object haschanged, at a higher magnification rate, and therefore it is possible toperform detailed observation. Furthermore, if the observationmagnification rate is fixed in advance to a high magnification rate,then it is only possible to observe a limited narrow range, and there isa possibility that change in the observation object which is of interestto the user cannot be detected, but in the present invention, it ispossible to detect change in the observation object over a broad range,and therefore the observation object can be observed reliably.

In the present embodiment, a biological specimen observation apparatuswhich performs cultivation and observation of biological specimens isdescribed as an example, but the present invention can also be appliedto an observation apparatus, such as a microscope, which carries outobservation of biological specimens only. Furthermore, by introducingimage analysis software according to the present embodiment, it ispossible to discover stem cell colonies in a liver, for example.Subsequently, it is possible to extract only good stem cell colonies bymicro image analysis of the stem cell colonies of the liver. Therefore,this is useful for hepatocellular management.

The series of processing described above can be executed by hardware orcan be executed by software. If the series of process is executed bysoftware, then a program which constitutes that software is installedfrom a storage medium to a computer equipped with dedicated hardware ora generic personal computer, for instance, which is capable of executingfunctions of various types by installing programs of various types.

This recording medium may be constituted by a magnetic disk, an opticaldisk, a magneto-optical disk or a semiconductor memory, or the like, onwhich a program is recorded and which is distributed in order to supplyprograms to a user, separately from a computer, or besides this, a ROM(Read Only Memory) or a HDD (Hard Disk Drive), or the like, which ispresented to a user in a previously installed state in a computer.

Furthermore, a program which causes the series of processing describedabove to be executed may be installed in a computer via a wired orwireless communications medium such as a local area network, theInternet, or digital satellite communications, by means of an interfacesuch as a router, modem, or the like, in accordance with requirements.

In the present specification, the steps which describe a program storedon a recording medium naturally include processing which is carried outin a time series according to the sequence in which it is listed, aswell as processing which is carried out in parallel or independently,rather than being carried out in a time series.

Furthermore, the embodiment of the present invention is not limited tothe embodiments described above and various modifications are possiblewithin a range that does not deviate from the essence of the presentinvention.

EXPLANATION OF REFERENCE NUMERALS

-   -   1: biological specimen observation apparatus; 11: first frame;        12: second frame; 15: culture vessel; 21: thermostatic chamber;        22: front face door; 23: front face opening; 24: inward/outward        conveyance port; 25: stocker; 26: vessel conveyance mechanism;        27: observation unit; 31: control unit; 32: display panel; 33:        observation information memory unit; 34: image analysis unit;        35: observation control unit; 41: transmission light        illumination unit; 42: epifluorescent illumination unit; 43:        specimen platform; 44: observation unit; 51: transmission light        LED; 52: transmission light optical system; 53 a to 53 c:        fluorescence LED; 54: fluorescence optical system; 55: objective        lens; 56: stage; 57: imaging unit; 58: image processing unit

1. A biological specimen observation apparatus which observes temporalchange in a biological specimen, comprising: macro image acquisitionmeans for acquiring a macro image by capturing an image of a macroregion of a broad range of the biological specimen, while time lapseobservation is performed; biological change region extracting means forextracting a biological change region, which is a region of change inthe biological specimen, from the acquired macro image; microobservation point setting means for registering a micro observationpoint corresponding to the extracted biological change region; microimage acquisition means for capturing an image of a micro region ofchange in the biological specimen identified at the micro observationpoint, while time lapse observation is performed; and judgment means forjudging whether or not biological change has continued in the biologicalchange region at the micro observation point, from the acquired microimage, wherein the micro observation point setting means updates theregistered micro observation point on the basis of a judgment result bythe judgment means.
 2. The biological specimen observation apparatusaccording to claim 1, wherein the micro observation point setting meanserases the registered micro observation point, when judgment is made bythe judgment means that there is no biological change.
 3. The biologicalspecimen observation apparatus according to claim 1 or 2, furthercomprising acceptance means for accepting registration or change of themicro observation point.
 4. The biological specimen observationapparatus according to any one of claims 1 to 3, further comprisingfirst tiled image generating means for generating a tiled image bypasting together a plurality of captured images obtained by capturingimages of the biological specimen, wherein the biological change regionextracting means extracts the biological change region from thegenerated tiled image.
 5. The biological specimen observation apparatusaccording to any one of claims 1 to 4, further comprising second tiledimage generating means for generating a tiled image, in a case where awhole range of the biological change region is not included in theacquired micro image, by acquiring a peripheral image obtained bycapturing images of a peripheral area of the micro observation point andpasting together the acquired peripheral image and micro image.
 6. Abiological specimen observation apparatus which observes temporal changein a biological specimen, comprising: macro image acquisition means foracquiring a macro image by capturing an image of a macro region of thebiological specimen, while time lapse observation is executed; microobservation point setting means for identifying a micro observationpoint where micro observation is to be performed in the macro region ofthe biological specimen, on the basis of the macro images of a timeseries obtained by the macro image acquisition means; and micro imageacquisition means for performing time lapse observation and acquiringmicro images of the biological specimen at the micro observation pointidentified by the micro observation point setting means.
 7. Thebiological specimen observation apparatus according to claim 6, furthercomprising: priority mode switching means for switching between a macroobservation priority mode which prioritizes macro observation by themacro image acquisition means and a micro observation priority modewhich prioritizes micro observation by the micro image acquisitionmeans; and control means for, in a case where there is overlap in anobservation schedule of the macro observation and micro observation,implementing control so that, first, acquisition of the macro image iscarried out and immediately after completion of this acquisition,acquisition of the micro image is carried out when the macro observationpriority mode has been selected, and implementing control so that,first, acquisition of the micro image is carried out and immediatelyafter completion of this acquisition, acquisition of the macro image iscarried out when the micro observation priority mode has been selected.8. A biological specimen observation apparatus which observes temporalchange in a biological specimen, comprising: macro image acquisitionmeans for acquiring a macro image by capturing an image of a macroregion of a broad range of the biological specimen, while time lapseobservation is performed; and micro image acquisition means foracquiring a micro image by capturing an image of a micro region in themacro region of the biological specimen, while time lapse observation isperformed, wherein temporal change in the biological specimen isanalyzed from the acquired macro image and micro image by the macroimage acquisition means and the micro image acquisition means.